Do you have an arcade panel configured or are you just testing out with a controller ?
The X button is mapped to the input_menu_toggle_btn action, so you can remove the line containing the mapping from the controller configuration file. The file is generated by Emulationstation in /opt/retropie/configs/all/retroarch-joypads, named after your controller. To be honest, I'd be more concerned about the Hotkey + B (reset content) combo, since it will reset the game, while Hotkey + X merely pauses it.
If you're using a keyboard, I think F1 toggles the menu.